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@ Method for forming sub Image data packet including data of sub image superimposed on main 
image, recording medium for recording sub image data packet, and image process apparatus. 

@ A sub image display control system is oonv 
posed of two sub Image display control portions 

3 that are a display region controt portion and a 
display timing control portion and a display 
^ output portion. A sub Image data packet is 
U) composed of three types of data that are a sub 
0> image header, sub image display data, and sub 
image control data. In this structure, display 
1^ spatial loss and display temporal loss can be 
^ remarkably reduced and sub images with wkle 
O applicabi% can be accomplished. 
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The present invention relates to an image proo- 
ess apparatus for superimposing a sub image such 
as a text on a nfiain image such as a TV image, a meth- 
od for forming a sub image data pacl^et, and a record- 
ing medium thereof. 

Conventionaity, translated texts for movies, con- 
trol and adjustment indications for TV remote control 
operations, and so forth are superimposed on main 
images by sub image process systems. 

The sub image display systems can be roughly 
categorized as character code system and bit map 
data system. 

In the character code type sub image process 
system, as shown in Fig. 27, character data (for ex- 
ample characters and patterns) stored in a character 
memory 60 are retrieved from a character generator 
81 as character code assigned to characters. The 
character code is transmitted to a sub image display 
portion 83. The character code is temporarily stored 
in a sub image frame buffer 84 and then required 
characters are displayed. In this system, dedicated 
hardware such as the character generator 81 is re- 
quired. Since data to be transmitted to a display por- 
tion is only character code, the amount of data to be 
transmitted and display time for display process can 
be reduced. 

However, since only prepared characters are dis- 
played, the applications for displaying sub images are 
limited. Thus, although this system is suitable for 
sound volume control screen of TV or the like, not for 
various mediums such as superimposed texts on 
movies. 

On the other hand, in the case of the bit map data 
system, since bit map data of sub images are directly 
transmitted to the display portion, the dedicated hard- 
ware that generate sub images conresponding to the 
character code is not required. In addition, since the 
shapes of sub images that can be displayed are not 
limited, the applications for displaying sub images are 
wide. 

However, the data rate of the bit map data system 
is much higher than that of the character code system. 
Thus, it can be said that the bit map data system has 
losses of data transmission time and display process 
time. 

In other words. In the bit map data system, each 
pixel should have sub image color data, sub image 
contour color data necessary for superimposing a sub 
image on a main image, and superimpose mixing ra- 
tion of the main image and sub image. Thus, the data 
rate of data to be transmitted to the display portion be- 
comes huge. 

In the bit niap data system, since pixel data with 
the same antount of data for one screen of a main im- 
age should be transmitted to the display portion, dis- 
play spatial loss takes place. Hereinafter, the data for 
one saeen refenred to as a frame. 

In additton, in any of the bit map system and the 



character code system, even if the shape of a sub im- 
age that is being displayed does not change, sub inrv 
age data should be continuously transmitted at each 
display frame interval. Thus, in data transmission and 
5 display control of these system, there is temporal 
loss. 

Consequently, in the above-described conven- 
tional sub inf^ge process systems, the bit map data 
system is superior to the character code system in ef- 

10 fectiveness of sub images. However, the bit map data 
system has display spatial loss and temporal loss. 

The present invention is made to soh^e the above- 
described problems. An object of the present inven- 
tion is to provide a sub image data packet forming 

15 method, a recording medium, and an image process 
apparatus that can remarkably reduce display spatial 
loss and display temporal loss and accomplish sub 
images that have wide applicabilities. 

To accomplish the above object, the present in- 

20 vention is a method or a recording medium for forming 
a sub infiage data packet including data of a sub image 
superimposed on a main image, the sub image data 
packet comprising a sub image data portion for stor- 
ing sub image data for displaying sub image informa- 

25 tton, a header portion for storing sub image header 
data for assigning at least a display position, a display 
size, and a display color of the sub image data, and 
a control data portion for storing control data for 
changing the display positron, the display size, and 

30 the deplay coIcm' of the sub image data assigned by 
the sub image header data, pixel by pixel. 

According to the present inventton, the sub image 
data packet is composed of the sub image data por- 
tk)n, the header portk)n, and the control data portksn. 

35 With the sub image data packet, the display positton, 
the display size, and the display color of the sub inv 
age data portion can be changed for each display unit 
of the main image. Thus, losses in the data transmis- 
sion, the display control, and so forth can be reduced. 

40 The header portion includes information that rep- 
resents whether data is present in the sub image data 
portion and the control data portion. Corresponding 
to the information, the data structure In the sub image 
data packet can be changed. 

45 The data of the control data portion included in 
the sub image data packet is used as control data for 
the sub image data portion that is displayed when 
data is absent in the sub image data portion of the sub 
image data packet Thus, it is not necessary to trans- 

50 mit the same data. Consequently, display spatial loss 
and display temporal loss can be reduced. 

The display position, the display size, and the dis- 
play color are not controlled for data in the sub image 
data portion of the sub image data packet, but dis- 
ss played corresponding to predetermined default val- 
ues when data is absent in the control data portion of 
the sub image data packet. Thus, the size of the 
structure of one packet can be reduced. Consequent- 
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ly, temporal loss of data transmission can be reduced. 

It Is detenmined that the sub Image data packet is 
In a display clear mode and data of the sub Image data 
portion is cleared when data is absent in the sub im- 
age data portion and the control data portion. Thus, 
it Is not necessary to newly transmit sub image data 
portion. Consequently, display spatial loss and dis- 
play temporal toss can be reduced. 

The header portion includes at least sub image 
display position/size assigning data for assigning a 
display position and a display size of the sub image 
superimposed on the main image corresponding to 
the type of the main image, display color assigning 
data for assigning a display color of the sub image, 
compensating execution assigning data for assigning 
emphasis of contour of the sub image and so forth, 
sub image contrast assigning data for assigning a 
mixing ratio of the sub image and the main image, and 
sub image display start timing assigning data for as- 
signing from what display unit of the main Image the 
display of the sub image is started. Thus, with one 
packet the sub Image can be superimposed on the 
main image in various manners. 

The type of data of the sub image data portion in- 
cludes Information that represents data correspond- 
ing to field/frame of the main image. Thus, corre- 
sponding to the value of the infonmatk)n, the sub im- 
age data portion can be recorded. 

A data format of the control data portion is as- 
signed by a change start line number, the number of 
continuous lines, and the number of pixel change 
points on each line of a region of the sub image, a pix- 
el number at which a change starts on each change 
line, a color of the sub Image display data after an as- 
signed pixel, a color of a contoured pixel, and a mbcing 
ratio between the main image and the sub image be- 
ing assigned repeatedly for the number of change 
lines assigned. Thus, the data amount of the sub im- 
age data packet can be remarkably reduced. 

The end of continuatton of the change line Is de- 
termined by assigning predetermined code to a line 
number, the number of continuous lines, and the num- 
ber of pixel change points. Thus, with the packet, the 
end of the continuation of change lines can be detect- 
ed. 

When the sub image information of the sub inv 
age data portion is displayed, the display position, the 
display size, and the display color are assigned cor- 
responding to the header portion by the control data 
portion for each display unit of the main image. Thus, 
even if the conventional packet is transmitted, the 
sub image can be more variously superimposed on 
the main image than the related art reference. 

These and other objects, features and advantag- 
es of the present invention will become more appa- 
rent in light of the following detailed description of a 
best mode embodiment thereof, as illustrated in the 
accompanying drawings. 



Fig. 1 is a schematic diagram showing a con- 
struction of a sub image output process system 
according to an embodiment of the present inven- 
tion: 

5 Fig. 2 is a table showing a data structure of a sub 

Image header according to the embodiment; 
Fig. 3 Is a schemata diagram showing a nK)de in 
which a sub image channel has all data according 
to the embodiment; 

10 Fig. 4 Is a schematic diagram showing a mode in 
which a sub image channel is composed of a sub 
\n\age header and sub image display data ac- 
cording to the embodiment; 
Fig. 5 is a schematic diagram showing a mode in 

15 which a sub image channel Is composed of a sub 
image header and sub Image control data ac- 
cording to the embodiment; 
Fig. 6 is a schematic diagram showing a mode in 
which a sub image channel is composed of only 

20 a sub Image header, 

Fig. 7 is a schematic diagram showing a display 
example of characters on a display frame accord- 
ing to the embodiment; 

Fig. 8 is a schematic diagram showing a con- 
25 struction of a color palette portk}n according to 
the embodiment; 

Fig. 9A is a schematic diagram showing sub Inrv 
age contour compensating information according 
to the embodiment; 
30 Fig. 9B is a schematic diagram for explaining a 
contour forming method according to the embodi- 
ment; 

Fig. 10 is a schematic diagram showing a con- 
struction of a mixing process system according to 
35 the embodiment; 

Fig. 11 is a conceptual schematic diagram show- 
ing sub image display timing in NTSC system ac- 
cording to the embodiment; 
Fig. 12 is a conceptual schematic diagram show- 
40 ing sub Image display clearing timing in NTSC 
system according to the embodiment; 
Fig. 13 is a table showing the relation between 
display data order and display positions accord- 
ing to the embodiment; 
45 Fig. 14 Is a schematic diagram showing the rela- 
tion between sub image display data and a dis- 
play image in frame mode A (non-interlace mode) 
according to the embodiment; 
Fig. 15 is a schematic diagram showing the rela- 
50 tion between sub image display data and a dis- 
play image in frame mode B (interlace mode) ac- 
cording to the embodiment; 
Fig. 16 is a schematic diagram showing the rela- 
tion between sub image display data and a dis- 
ss play image in field nfK>de according to the em- 
bodiment; 

Fig. 17 is a schematic diagram showing a struc- 
ture of sub image control data according to the 
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embodiment; 

Fig. 18 is a schematic diagram showing a struc- 
ture of frame control data according to the env 
bodiment; 

Fig. 1 9 is a schematic diagram showing an exam- 
ple of a sub image display pattern according to 
the embodiment; 

Fig. 20 is a schematic diagram showing an exanv 
pie of frame control data according to the em- 
bodiment; 

Fig. 21 Is a schematic diagram showing reserved 
bits of a bit structure in binary notation according 
to the embodiment; 

Fig. 22 Is a schematic diagram showing changes 
of sub image color, contour compensating color, 
and contrast ratio on line "4" according to the em- 
bodiment; 

Fig. 23 is a schematic diagram showing a change 
start pixel number in binary notation at position C 
according to the embodinnent; 
Fig. 24 is a table showing the data of Fig. 20 in 
hexadecimal notation; 

Fig. 25 is a schematic diagram showing a con- 
struction of a sub Image process system accord- 
ing to the embodiment; 

Fig. 26 Is a schematic diagram showing a timing 
chart on which frame control data is processed 
according to the embodiment; and 
Fig. 27 Is a schenuitic diagram for explaining a 
conventional character code system. 
Next, with reference to the accompanying draw- 
ings, an embodiment of the present invention will be 
described. 

Fig. 1 is a schematic diagram showing a con- 
struction of a sub image output process system ac- 
cording to the embodiment of the present invention 
and a structure of image infonmation such as sub inv 
age data packet reproduced from an optical disc or 
the like by the system. Fig. 2 is a table showing data 
structure of a sub image header in the sub Infiage data 
packet 

As shown in Fig. 1 , the sub image output process 
system 1 comprises two sub image display control 
portions and a display output portion 4. The sub Inv 
age display control portions are a display region con- 
trol portion 2 and a display timing control portion 3. 

A sub image data packet 5 Is composed of three 
types of data that are a sub image header 6, sub inv 
age display data 7, and sub image control data 8. The 
sub Image header 6 is control information for assign- 
ing the display region on the main image frame and 
the display manner of the sub image display data 7. 
The data arrangement and data amount of each data 
in the sub image data packet 5 are not basically linrv 
ited. 

As shown In Fig. 2, the sub image header 6 is 
composed of various control information and so forth 
that are for example a sub innage channel forming 



mode that Is sub image fonning informatton (paranne- 
ter SPCSTR), a mixing ratio of an sub Image and a 
main Image that is a sub image contrast (parameter 
SPCONT), a sub image display start timing (parame- 

5 ter SRCDST), a sub image display end timing (para- 
meter SPCDEN), a sub in^age display start position 
(X, Y), display size such as display width and display 
height (parameter SPCSIZE). sub image color infor- 
mation (parameter SPCINFO), sub image contour 

10 compensating information (parameter SPADJINFO), 
and sub image control data start address (parameter 
SPCDADR). 

When the sub image output process system 1 
reads the sub image data packet 5, it separates three 

15 types of data that are for example the sub image 
header 6, the sub image display data 7, and the sub 
image control data 8 from the sub image data packet 
5 corresponding to each control information. 

Thereafter, the sub image header 6 is supplied to 

20 the display region control portton 2 that controls the 
diSF^ay region. The sub Image control data 8 is sup- 
piled to the display timing control portion 3. The sub 
image display data 7 is controlled by the display con- 
trol portions 2 and 3 and supplied to the display output 

25 portion 4. 

The display region control portion 2 controls the 
display positk)n on the main image frame and the dis- 
play manner of the sub image display data 7 corre- 
sponding to the received sub image header 6. The 

30 display timing control portion 3 perfomns a temporal 
display control such as color change of a sub image 
and a change of mixing ratio of a main image and a 
sub image in the condition that the display time inter- 
val of one main Image frame is one control unit cor- 

35 responding to the sub Image control data 8. The dis- 
play output portion 4 is controlled by the two display 
control portbns 2 and 3 and the sub image display 
data 7 is displayed and output from the system. 
Since the sub image output process system 1 dis- 

40 plays the next sub image screen corresponding to 
each data of the data packet on the present sub im- 
age screen, unless the display region of the sub inv 
age is changed or part of the sub image display data 
7 is changed, the same sub Image screen that is dis- 

45 played on for example a plurality of main Image 
frames is displayed corresponding to the first sub im- 
age data packet 5. 

Thus, It Is not necessary to supply pixel data for 
frames to be displayed as the sub image display data 

50 7 to the sub image output process system 1 . In other 
words, the sub image output process system 1 satis- 
factorily operates with display data amount for one 
frame. Consequently, the data amount necessary for 
images that change can be remarkably reduced. 

55 As described above, the sub Image header 6 
shown in Fig. 2 Includes the sub image channel form- 
ing mode (parameter SPCSTR) that represents 
whether or not the sub image display data 7 and the 
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sub image control data 8 are present in the sub image 
data packet 5. The sub image channel forming mode 
has four sub modes that are "100", "101", "110", and 
•111". 

The sub Image output process system 1 reads 
the sub image data packet 5. When the sub image 
output process system 1 reads the value of the para- 
meter SPCSTRf rom the sub image header 6 and de- 
termines whether or not to perform the display control 
for the sub image. 

When "111" is assigned to the parameter 
SPCSTR (in the case of the first mode), as shown In 
Fig. 3, the sub image channel is composed of the sub 
image header 6, the sub image display data 7, and the 
sub image control data 8. Since this mode is normal 
mode in which all data is present, the description 
thereof is omitted. 

When "110" is assigned to the parameter 
SPCSTR (in the second mode), as shown in Fig. 4, 
the sub image channel is composed of the sub image 
header 6 and the sub Image display data 7. Thus, the 
sub image control data 8 Is absent In the sub image 
data packet 5. 

As shown in Fig. 4, in the second mode, the sub 
Image output process system 1 read the sub image 
header 6 and determines that the sub image control 
data 8 is absent corresponding to the sub Image fornv 
ing Information of the sub image header 6. Thus, the 
sub image output process system 1 does not perform 
the display control operation. 

When "101" is assigned to the parameter 
SPCSTR (In the third mode), as shown in Fig. 5, the 
sub image channel is composed of the sub image 
header 6 and the sub Image control data 8. Thus, the 
sub image display data 7 is absent. 

In the third mode, as shown in Fig. 5, the sub im- 
age output process system 1 reads the sub image 
header 6 corresponding to the sub image forming In- 
formation of the sub image header 6 and detemnines 
that the sub image display data 7 Is absent Thus, the 
sub Image output process system 1 does not change 
the display image. In this case, the sub image output 
process system 1 continues to display the image cor- 
responding to predetermined default values of the dis- 
play position, display size, and display color informa- 
tion of the sub Image. When and where the state of 
the display image changes, new color Information is 
assigned. In this case, in addition to the sub Image 
display data, color information that is newly assigned 
is fonfned as a packet of frame control data. Since the 
position of the frame control data can be assigned, 
pixel by pixel, the sub image color control can be per- 
formed with the same accuracy as the conventional 
bit map data system. 

When "100" is assigned to the parameter 
SPCSTR (in the fourth mode), as shown in Fig. 6, the 
sub image channel is composed of only the sub Im- 
age header 6. In this mode, the sub image screen Is 



cleared. In this mode, the sub image display data 7 
and the sub image control data 8 are absent. 

In the fourth mode, as shown in Fig. 6, the sub inv 
age output process system 1 reads the sub image 

5 header 6 corresponding to the sub image fonnrting In- 
fonmation of the sub image header 6 and determines 
that the sub image display data 7 and the sub image 
control data 8 are absent Thus, the sub image output 
process system 1 clears the sub image screen con-e- 

10 sponding to information of other sub Image data. 

Consequently, when temporal control of the sub 
image control data 8 is not required or when the sub 
image display data 7 is the same as the data that has 
been displayed and only the display control is re- 

15 quired, even if data to be present in the sub image 
data packet 5 (respective data are omitted), the sub 
image display operation can be performed without a 
problem. 

Next, with reference to Figs. 7 to 12, the functions 
20 of the parameters in the sub image header 6 will be 
described. 

First the sub image display position/display size 
will be described. 

The sub image display position/display size Is 
25 control infonnation that represents the display posi- 
tion and display region of the sub Image display data 
on the frame. 

The display position and display region on the 
frame are represented by the sub image display pos- 
30 ition/display size data as a bit structure such as [b79 
X coordinate b64J, [b63 Y coordinate b48], [b47 Re- 
served b32], [b31 Display width b16], and [b15 Dis- 
play height bO]. 

In this case, as shown in Fig. 7, dot characters as 
35 in "ABX" are displayed on the display frame 10. In Fig. 

7, t represents "1 ' of the sub image data. 0 repre- 
sents "0" of the sub image data. X represents X co- 
ordinate that is the display start position In horizontal 
direction (dot by dot). Y represents Y coordinate that 
^ Is the display start position in vertical direction (line by 
line). W represents display width that Is the width of 
the display region in horizontal region (dot by dot). H 
represents display height that is the width of the dis- 
play region in vertical direction (line by line). 

The parameter SPCINFO of the sub image color 
information designates a color of a portion of which 
the sub Image data is "1". The parameter SPCINFO 
of the sub Image color information is not direct color 
data, but a selection address that assigns one of col- 
ors on the color palette 1 1 (palette address) as shown 
in F^. 8. The selected color conresponding to the se- 
\ecX\on address (palette address) and the display data 
are ANDed by a gate 12 and output as the sub image 
display color. The sub image color information data is 
represented as a bit structure such as [b7 Not used 
b4] and [bS Sub Image color bO]. 

As shown In Fig. 9A, in the sub image contour 
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compensating infomiation (parameter SPADJINFO), 

* represents "1" of the sub image data. □ represents 

"0" of the sub image data. @ represents the contour 

of * When the contour compensation is performed, 
a portion of which the sub image display data is "1" is 
contoured in vertical, horizontal, and oblique direc- 
tions for the number of dots assigned. In addition, the 
sub image contour compensating information (para- 
meter SPADJINFO) also assigns a color of a portion 
of which the sub image display data is T. 

As shown in Fig. 9B, in the contour forming meth- 
od corresponding to the sub image contour compen- 
sating information (parameter SPADJINFO). when 
the system display portion scans up to a dot unit X of 
a display region, if there Is a region of which the sub 
image data is "r in a region of a dot width A in hori- 
zontal and vertical direction (this region is denoted by 
0 In Fig. 9B), as long as the sub image data at the dot 
position X Is not "1", the dot position X Is contoured. 
When this operation is performed along dot positions 
of the entire display region, any portion of the display 
region can be contoured. 

The region of the contour compensation can be 
represented by the sub image contour compensating 
information data as a bit structure such as [b7 Exe- 
cution flag b6], [bS Contour compensating width b4], 
and [b3 Contour compensating color bO]. The execu- 
tion flag selects whether or not to contour a portion 
of which the sub image data is "1". The contour conv 
pensating width represents the width of the contour 
to be compensated, dot by dot. The contour compen- 
sating color represents a display color of the contour 
compensating portion. The parameter (SPADJINFO) 
is not direct color data, but a selection address that 
assigns one of predetermined colors. The basic con- 
ception of the sub image contour compensating infor- 
mation Is the same as that of the sub image color in- 
formation. 

When a portion of which the sub image data is "1 " 
is displayed, the parameter (SPCONT) of the sub inv 
age contrast represents the mixing ratio of a main im- 
age and a sub image. The main image and the sub inr)- 
age are mixed by logically adding the main image 
component and sub image contrast corresponding to 
the sub Image contrast This addition can be repre- 
sented by the following equation. 

Output of display image = main image component 
+ sub image component = (255 - value of sub image 
contrast) -i- 255 x image data (value of sub image 
contrast + 255) x sub image color information 
However, this equation is satisfied only when the sub 
inrtage data s "1". When the sub Image data is '0", the 
output of the display image is equal to the main image 
component 

Fig. 10 shows a logical system (mbcing process 
system) that performs the mixing operation. The log- 



ical system comprises selectors 20 and 21, dividing 
devices 22 and 23, a subtracting device 24, multiply- 
ing devices 25 and 26, and an adding device 27. 

In the mixing process system, the mixing ratio in 
the case that the sub image data is "r is substantially 
different from the mixing ratio in the case that the sub 
image data Is "0". When the sub image data is "1", the 
sub image contrast is used as a substantial mixing ra- 
tio. When the sub image data is "0". "OOH" Is used as 
a substantial mixing ratio. 

Thus, the sub image can be faded In and out 

The parameters (SPCDSTand SPCDEN) for the 
sub image display start timing/display end (clear) tim- 
ing control the display intervals of sub images. 

The parameter (SPCDST) for the sub image dis- 
play start timing control starts the display of the sub 
Innage display data at a frame number represented by 
a frame counter. 

For example, as shown In Fig. 11, in the case of 
the NTSC system, the display count of the sub image 
counts from frame "3" at an I picture outputf rame and 
continues counting (incrementing) up to frame "30" 
that is two frames before the next I picture. 

The sub image that is displayed at the frame 
number of the frame count is continuously displayed 
until the next sub image display start point takes 
place or the sub image Is cleared by the sub Image 
clear timing data. In Fig. 11 , since the sub image dis- 
play timing data starts with frame "6" and ends with 
frame "4", the first sub image is displayed while the 
display frame count counts from frame "6" to frame 
"4". 

The sub image display data is cleared at the as- 
signed frame number of the frame count correspond- 
ing to the sub image dear timing data. 

As shown in Fig. 12, the sub image display data 
Is cleared at the frame "28". 

Next, with reference to Figs. 13 to 16, the struc- 
ture of sub image display data and a display Image 
will be described. Fig. 13 shows the relation between 
a display data order and display positions thereof. 
Fig. 14 is a schematic diagram showing an example 
of the relation between sub image display data and a 
display image in frame mode A(non-interlace nK>de). 
Fig. 15 is a schematic diagram showing an example 
of the relation between sub image display data and a 
display image in frame mode B (interlace mode). Fig. 
1 6 is a schematic diagram showing an example of the 
relation between sub image display data and a dis- 
play Image in field mode. 

In Fig. 13. sub image display data is displayed 
from left to right of display pixels and from top to down 
of display lines of the display region. 

The display method and record method of the sub 
image display data can be categorized as frame 
mode (shown in Figs. 14 and 16) and field mode 
(shown in Fig. 16). When the sub image display data 
is recorded in the sub Inr^ge data packet, the record 
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mode of the sub Image display data Is also recorded. 
When the reproducing system displays data, It de- 
tects the record wode and controls the display mode. 

In the frame mode A (non-inter lace mode) shown 
in Fig. 14, the data order of the sub innage display data 
accords with the scan order of the display Image. On 
the other hand» in the frame mode B (Interlace mode) 
shown in Fig. 15, although the data order of the sub 
image display data does not accord with the scan or- 
der of the display image, the data amount of the sub 
image display data accords with the data amount of 
the display image. 

As shown in Fig. 16, it is clear that even if the 
same region is displayed, the data amount in the field 
mode is half the data amount in the frame mode. 

Next, with reference to Fig. 17. the structure of 
the sub Image control data will be described. 

Referring to Fig. 17, sub image control data 30 is 
composed of a sub image control data header 31 and 
frame control data 32. 

in the sub Image control data header 31 , display 
frame control data start addresses (CDADRF 1 to 30) 
33 are present corresponding to frames of the main 
image 1 . On the other hand, in the frame control data 
32, the frame display control data 34 are present cor- 
responding to display frames. The display frame con- 
trol data start addresses (CDADRF 1 to 30) 33 repre- 
sent relative positions from the beginning of the sub 
image control data header 31 of frame control data. 

Next, the structure and function of the display 
frame control data start addresses will be described. 
In the sub Image reproducing system, the display 
frame control data start addresses (CDADRF 1 to 30) 
33 control the sub Image display, frame by frame. 

Thus, the sub image reproducing system gener- 
ates the start address of the frame control data cor- 
responding to the display frame control data start ad- 
dress 33 of a frame number to be controlled. With the 
start address, the sub image reproducing system ac- 
cesses the frame control data. The display frame 
control data start address 33 represents the start pos- 
ition of each frame control data that Is the number of 
relative bytes from the sub image control data head- 
er. For example, CDADRF 1 represents the start pos- 
ition of the control data of frame 1. Likewise, 
CDADRF 2 represents the start position of the control 
data of frame 2. And so on. 

Next, with reference to Figs. 1 8 to 20, the struc- 
ture and function of the frame control data 32 will be 
described. Fig. 18 shows a structure of frame control 
data. Fig. 19 shows an example of a sub innage dis- 
play pattern. Fig. 20 shows an example of frame con- 
trol data. 

As shown in Fig. 18, the frame control data 32 for 
one frame is composed of a change line number 35 
and change start pixel numbers 36. 

The change line number 35 represents from what 
line of a display frame the contour corresponding col- 



or, the sub Image color, and the contrast are control- 
led, how many times they are changed on the line, 
and how many lines they are changed. 

The change line number 35 is represented as a 
5 bit structure such as [b31 Reserved b28], [b27 
Change line number b16], [b15 Number of pixel 
change points b12], and [b11 Number of continuous 
lines bO]. 

The change start pixel number 36 represents a 
10 pixel position at which the contour compensating col- 
or, the sub image color, and the contrast are changed 
and a contour compensating color, a sub pixel color, 
and a contrast that have been changed. The change 
start pixel numbers continue for the number of 
15 change points corresponding to the change line num- 
ber. The change start pixel number is represented as 
a bit structure such as [b31 Reserved b28], [b27 
Change pixel number b16], [b15 Sub image color af- 
ter change b1 2], [b1 1 Contour compensating color af- 
20 ter change b8], and [b7 Contrast ratio after change 
bO]. 

In the case of the above-described bit structure, 
the sub image display pattern is represented by a 
combination of the change line number 35 and the 

25 change start pixel numbers 36. 

For example, as shown In Fig. 19, In the case of 
the bit structure of the change line number, pixel 
changes start from line "4" to line "11". Since line 
changes start from line "4", the change line number 

30 35 is '4" and there are three pixel changes at posh 
tions A, B, and C, the number of pbcel change points 
is "3" and pbcel change state continues to line *ir. 
Thus, the number of continuous lines Is ("11" - "4" = 
•7"). 

35 At the beginning of the frame control data 32 of 
the frame, a change line number that represents that 
a change start line is "4" is present. 

Since pixel changes on each line from line "4" to 
line "ir are the same in the order of positions C, B, 

40 and A, change start pbcel numbers at positions C, B, 
and A are arranged from the line following the first 
change line. This relation is shown in Fig. 20(a). 

The number of pixel change points, position, sub 
in^ge color, contour compensating color, and con- 

45 trast ratio on line "12" are different from those on line 
"14". Thus, for each line, a change line number and a 
change start pixel number should be formed. The 
number of change continuous lines of each of change 
line numbers "12" and "14" is "0" because there are 

50 no continuous lines. 

On line "12", there are two pbcel change positions 
at positions D and E. The change line number that 
represents the number of continuous line numbers is 
"0" Is followed by change start pbcel numbers at pos- 

55 itions D and E as shown in Fig. 20(b). 

On line "14", there are four pbcel changes at pos- 
itions D, C. E, and A. Thus, a change line number that 
represents that the number of continuous lines is "0" 



8 



13 



EP0 677 954 A2 



14 



and that the number of pixel change points is is fol- 
iowed by change start pixel numbers at positions D, 
and A as shown in Fig. 20(c). Since there are no 
pixel changes on line "13", the frame control data 32 
Is not formed. 5 

On line "3", regions of up to position C on lines "4" 
to "ir, line "13", and regions of up to position D on 
lines "12" to "14", a sub image with a sub image color, 
a contour compensating color, and a contrast conre- 
spending to the sub image control data header 31 are io 
displayed Instead of frame control data 32. The order 
of the change line number 35 and the change start 
pixel numbers 36 of the frame control data 32 shown 
in Figs. 18 to 20 should accord with the scan direction 
of a frame to be displayed. is 

As shown in Fig. 20(d), in the frame control data 
32, a change line number 35 that is defined by a pre- 
detenmined pattern and that follows the last change 
start pixel number 36 is formed so that a change start 
pixel number 36 on a continuous line represents the 20 
end of the data structure. The predetermined pattern 
of the end code is for example a bit structure 
"FFFFOOOOH" (hexadecimal notation). 

When the reserved bit [b31 Reserved b28] in the 
bit structure is "OH", the change tine number 35 is 2S 
represented in binary notation as follows (see Fig. 
21). 

-00000000000011000010101011111111". 
The change line number 35 Is represented in hexade- 
cimal notation as follows. 30 
"CSLN = 00043007H" 

Next, the structure of the change start pixel nunrv 
ber 36 on for example line "4" will be described. 

Fig. 22 shows changes of sub image colors, con- 
tour compensating colors, and contrast ratios. In this 35 
case, the pixel change points are present at positions 
C, B, and A in line direction. First, the pixel change 
start point at position C is represented corresponding 
to the example of bit structure. Since positbn C is 
present at dot 12 from the left edge of the line, the 40 
change pixel number is "12". Referring to Fig. 12, the 
sub image color, the contour compensating color, and 
the contrast ratio that are present on the right of pos- 
ition C are "2H". "AH", and "FFH". respectively. 

The change start pixel number at position C is 45 
represented in binary notation as follows (see Fig. 
23). 

•00000000000001 00001 1 0000000001 1 1 ". 
The change start pbcel number is represented in hex- 
adecimal notation as follows. so 
■CSPN = 000C2AFFH". 
At position C, although the contour compensating 
color does not change, the right side portion (on the 
line in scan direction) of position C should be written 
for the changed contour compensating color. This ap- 55 
plies to the other sub image colors and contrast ra- 
tions. 

Position B is present at dot 15 and only the sub 



image color changes from '2H" to "IN". The change 
start pixel number at position B can be represented in 
hexadecimal notation as follows. 

"CSPN (at position B) = 000F1AFFH" 

that is changed from 

"CSPN (at position A) = 000C2AFFH". 
Thus, it Is clear that "02" is changed to "F1". Likewise, 
Position A is present at dot 1 8 and the sub image color 
and contour compensating color change to "5H" and 
'4H", respectively. Thus, the change start pbcel num- 
ber at position A can be represented in hexadecimal 
notation as follows. 

"CSPN (at position A) = 001254FFH" 

that is changed from 

'CSPN (at position B) = 000F1AFFH" 
Thus, it is dear that "OFAI" is changed to "1254". 
Consequently, the data shown in Fig. 20(a) can be 
represented in hexadecimal notation as shown in Fig. 
24. 

Next, with reference to Figs. 25 and 26, the con- 
struction of the sub image process system and oper- 
ating timing will be described. Fig. 25 is a block dia- 
gram showing the sub image process system. Fig. 26 
is a timing chart of a process for the frame control 
data. 

As shown in Fig. 25, the sub image process sys- 
tem comprises a change line number data register 50, 
a line counter 51, comparators 52 and 53, a change 
start pixel data read cycle control portion 54, a 
change start pixel number data register 55, a change 
pixel number register 56, a dot counter 57, a register 
read control portion 58, a sub image color register 59, 
a contour compensating color register 60, and a con- 
trast ratio register 61. 

The change line number data register 50 holds a 
change line number corresponding to frame control 
data. The line counter 51 counts a line number. The 
comparator 52 compares a change line number with 
a line count value. When they match, the comparator 
52 outputs a match signal. The change start pbcel data 
read cycle control portion 54 receives the number of 
pixel change points, the match signal (received from 
the comparator 52), a horizontal (H) blanking signal, 
and so forth and controls the holding timing of the 
change start pixel number data. The change start pix- 
el number data register 55 holds positton data at the 
control timing of the change start pixel data read cycle 
control portion 54. The change pixel number register 
56 holds a change pbcel number. The comparator 53 
compares the count value of the dot counter 57 with 
the value of the change pbcel number register. When 
they match, the comparator 53 outputs a match signal 
to the register read control portion 58. The register 
read control portion 58 control the data hold timing of 
each of the sub image color register 59, the contour 
compensating color register 60, and the contrast ratio 
register 61 corresponding to the match signal re- 
ceived from the comparator 53. 
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In the sub image process system, the frame con- 
trot data is processed as follows. 

In other words, as shown In Fig. 26, when the line 
count number counted by the line counter 51 of the 
display portion matches the change line number In 5 
the change line number data of the frame control 
data, the comparator 52 outputs the match signal to 
the change start pixel data read cyde control portion 
54. Thus, the change start pixel data read cycle con- 
trol portion 54 starts the read cyde. 

The change start pixel data read cyde control 
portion 54 stores change start pixel numbers neces- 
sary for changing pixels on a line to a process portion 
(namely, each change start pixel number data register 
55) in the first horizontal (H) blanking interval. Alter- 
natively, the change start pixel data read cyde control 
portion 54 stores the change start pixel numbers to 
each change start pixel number data register 55 until 
change line number data that represents end code 
takes place. 

At this point, the horizonal dot counter 57 is incre- 
mented by "1" for the line. When the comparator 53 
has determined that the dot count value matches the 
change pixel number of the change start pixel number 
data, the comparator 53 outputs the match signal to 
the register read control portion 58. 

The register read control portion 58 holds a sub 
image color, a contour compensating color, and a 
contrast ratio that have been changed in the process 
portion (namely, the sub image color register 59. the 
contour compensating color register 60, and the con- 
trast ratio register 61 ). The data held in the sub image 
color register 59, the contour compensating color reg- 
ister 60. and the contrast ratio register 61 are used as 
sub Image display control data until next pixel change 
point takes place. 

Although the sub image process system per- 
forms the above-described operation whenever it 
scans each line, when the number of continuous lines 
in the change line number is other than "0*. after the 
count line number matches the change line number of 
the change line number data, while the tines corre- 
sponding to the number of continuous lines are being 
scanned, the same change start pixel number data is 
read and the same sub image display control is re- 
peated. 

According to the embodiment of the present in- 
vention, since data amount is remarkably reduced by 
the simple sub image data packet system and the de- 
gree of flexibility of the representation of a sub Image 
is provided by the bit map data system, display spatial 
loss and display temporal loss of the sub image data 
can be reduced and the applications of the sub image 
can be widened. 

According to the sub image data packet system, 55 
since data that assigning a display region of a sub inn- 
age is provided and other than the display region is 
not displayed, display spatial loss of which all data for 



one frame is transmitted to the display portion can be 
remarkably reduced. In an assigned display region, 
only information of shapes of sub images to be dis- 
played are sub image display data. Thus, color infor- 
mation data, contour color data, and contrast data 
(these data are referred to as color information) that 
are required for each pixel can be reduced. In addi- 
tion, since the sub image display data is bit map data, 
the shapes of sub images can be freely displayed. 

Default values are assigned for the color informa- 
tion of sub images. When and where the state of a sub 
image changes, new color informatton is assigned. 
The color information that is reassigned is formed as 
a packet of frame control data independent from sub 
image display data. The position of the frame control 
data can be assigned, pixel by pbcel. Thus, the sub im- 
age color can be controlled with the same accuracy 
as the conventional bit map data system. Since most 
color information of a normal sub image does not 
change, pbcel by pixel, the color information is not as- 
signed, pixel by pixel. Thus, the color information data 
(frame control data) can be remarkably reduced. 

The sub image display data is continuously out- 
put from the display portion in a plurality of frame in- 
tervals unless the shape of the sub image changes. 
In these intervals, the color of the sub image is con- 
trolled by frame control data in the same sub image 
packet Thus, it is not necessary to continuously sup- 
ply the complete bit map data even if color Infonmation 
partly changes. Thus, display temporal loss of the sub 
image data can be remarkably reduced. 

As described above, according to the present in- 
vention, the display position, the display size, and the 
display color of the sub image data can change for 
each display unit of the main image. Thus, losses in 
data transmission and display control can be re- 
duced. 

The sub image can be freely displayed by the bit 
map data system. In addition, infonmation that as- 
signs a display region of the sub image Is provided so 
that data other than the region is not transmitted and 
displayed. Thus, since it is not necessary to transmit 
all pixel data for one frame to the display portion, dis- 
play spatial loss can be reduced. 

Consequently, display spatial loss and display 
temporal loss can be remarkably reduced and sub im- 
ages that have large flexibility can be accomplished. 

In the assigned display region, only information of 
shapes of sub images to be displayed Is treated as 
display data. The display color, the contour compen- 
sating color, and the sub image contrast that are re- 
quired for each pbcel are formed in a header portk)n. 
Thus, the amount of data to be transmitted can be re- 
duced. Since the display data portion is bit map data, 
the shapes of sub images can be freely displayed. 

Default values are assigned for the color infonma- 
tion of sub i mages. When and where the state of a sub 
image changes, new coHw informatbn Is assigned. 
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The color information that is reassigned is formed as 
a packet of frame control data Independent from sub 
image display data. The position of the frame control 
data can be assigned, pixel by pbcel. Thus, the sub im- 
age color can be controlled with the same accuracy 5 
as the conventional bit map data system. Since most 
color information of a normal sub image does not 
change, pixel by pixel, the color infonmation is not as- 
signed, pixel by pixel. Thus, the color information data 
(frame control data) can be remarkably reduced. io 

The sub image display data is continuously out- 
put from the display portion in a plurality of frame in- 
tervals unless the shape of the sub image changes. 
In these intervals, the color of the sub image is con- 
trolled by frame control data in the same sub image is 
packet Thus, it is not necessary to continuously sup- 
ply the complete bit map data even if color information 
partly changes. Thus, display temporal loss of the sub 
Image data can be remarkably reduced. 

Although the present invention has been shown 20 
and described with respect to a best mode embodi- 
ment thereof, it should be understood by those skilled 
in the art that the foregoing and various other 
changes, omissions, and additions in the form and de- 
tail thereof may be made therein without departing 25 
from the spirit and scope of the present invention. 



Claims 

30 

1 . A method for forming a sub i mage data packet in- 
cluding data of a sub image superimposed on a 
main image, the sub image data packet compris- 
ing: 

a sub image data portion for storing sub 35 
image data for displaying sub image information; 

a header portion for storing sub image 
header data for assigning at least a display posi- 
tion, a display size, and a display color of the sub 
image data; and 40 

a control data portion for storing control 
data for changing the display position, the display 
size, and the display color of the sub image data 
assigned by the sub image header data, pixel by 
pixel. 45 

2. The method as set forth in claim 1 , wherein said 
header portion includes information that repre- 
sents whether data is present in said sub image 
data portion and said control data portion. so 

3. The method as set forth in claim 1 , wherein data 
of said control data portion included in the sub inv 
age data packet Is used as control data for said 

sub image data portion that is displayed when 55 
data is absent in said sub image data portion of 
the sub image data packet 



4. The method as set forth in daim 1 , wherein the 
display position, the display size, and the display 
color are not controlled for data in said sub image 
data portion of the sub image data packet, but 
displayed corresponding to predetermined de- 
fault values when data is absent in said control 
data pc^ion of the sub image data packet 

5. The method as set forth in claim 1, wherein it is 
determined that the sub image data packet Is in 
a display dear mode and data of sakl sub image 
data portion is deared when data is absent in said 
sub image data portion and saM control data por- 
tion. 

6. The method as set forth in daim 1, wherein said 
header portion indudes at least: 

sub image display position/size assigning 
data for assigning a display position and a display 
size of the sub image superimposed on the main 
image corresponding to the type of the main im- 
age; 

display color assigning data for assigning 
a display color of the sub image; 

compensating executbn assigning data 
for assigning emphasis of contour of the sub im- 
age and so forth; 

sub image contrast assigning data for as- 
signing a mixing ratio of the sub image and the 
main image; and 

sub image display start timing assigning 
data for assigning from what display unit of the 
main image the display of the sub image is start- 
ed. 

7. The method as set forth in daim 1 . wherein the 
type of data of said sub image data portion in- 
cludes information that represents data corre- 
sponding to field/frame of the main image. 

8. The method as set forth In daim 1, wherein sakJ 
control data portion indudes: 

control data corresponding to the display 
unit of the main image; and 

sub Image control data header for assign- 
ing the start position of the control data. 

9. A method for generating a sub image correspond- 
ing to a sub image data packet induding data for 
the sub Image superimposed on a main Image, 
the sub image data packet comprising: 

a sub image data portion for storing sub 
Image data for displaying sub image information; 

a header portion for storing sub image 
header data for assigning at least a display posi- 
tion, a display size, and a display color of the sub 
image data; and 

a control data portton induding control 
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data and a sub image control data header, the 
control data being adapted for changing the dis- 
play position, the display size, and the display 
color of the sub image data assigned by the sub 
image header data, pixel by pixel, the sub image 5 
control data being adapted for assigning the start 
position of the control data, 

wherein a data format of said control data 
portion is assigned by a change start line number, 
the number of continuous lines, and the number io 
of pixel change points on each line of a region of 
the sub image, a pixel number at which a change 
starts on each change line, a color of the sub inrv 
age display data after an assigned pixel, a color 
of a contoured pb(el, and a mixing ratio between is 
the main image and the sub image being as- 
signed repeatedly for the number of change lines 
assigned. 

10. The method as set forth in claim 9, wherein the 20 
end of continuation of the change line is deter- 
mined by assigning predetenmined code to a line 
number, the number of continuous lines, and the 
number of pixel change points. 

25 

11. A recording medium for recording a sub image 
data packet of sub image information supehnv 
posed on a main image, the sub image data pack- 
et comprising: 

a sub Image data portion for storing sub 30 



Innage data for displaying sub Image infonrnation; 

a header portion for storing sub image 
header data for assigning at least a display posi- 
tion, a display size, and a display color of the sub 
image data; and 

a control data portion for storing control 
data for changing the display position, the display 
size, and the display color of the sub image data 
assigned by the sub image header data, pixel by 
pixel. 

12. The recording medium as set forth in claim 11. 
wherein said header portion includes infomiation 
that represents whether data is present in said 
sub image data portion and said control data por- 
tion. 

13. The recording medium as set forth in claim 11, 
wherein data of said control data portion included 
in the sub Image data packet is used as control 
data for said sub image data portion that is dis- 
played when data is absent in said sub image 
data portion of the sub image data packet. 

14. The recording medium as set forth in daim 11, 
wherein the display position, the display size, and 
the display color are not controlled for data in said 
sub image data portton of the sub image data 



packet, but displayed corresponding to predeter- 
mined default values when data is absent in said 
control data portk)n of the sub Image data packet 

15. The recording medium as set forth in claim 11, 
wherein it is detemnined that the sub image data 
packet is in a display clear mode and data of said 
sub image data portion is cleared when data is 
absent in said sub image data portion and said 
control data portton. 

16. The recording medium as set forth in claim 11, 
wherein said header portion includes at least: 

sub image display position/size assigning 
data for assigning a display position and a display 
size of the sub image superimposed on the main 
image corresponding to the type of the main inv 
age; 

display color assigning data for assigning 
a display color of the sub image; 

compensating executton assigning data 
for assigning emphasis of contour of the sub im- 
age and so forth; 

sub image contrast assigning data for as- 
signing a mixing ratio of the sub image and the 
main image; and 

sub image display start timing assigning 
data for assigning from what display unit of the 
main image the display of the sub image is start- 
ed. 

17. The recording medium as set forth in claim 11, 
wherein the type of data of said sub image data 
portion includes information that represents data 

35 corresponding to field/frame of the main image. 

18. The recording medium as set forth in claim 11, 
wherein said control data portion includes: 

control data corresponding to the display 
40 unit of the main Image; and 

sub image control data header for assign- 
ing the start position of the control data. 

19. An image process apparatus for generating a sub 
45 image corresponding to a sub image data packet 

including data for the sub image and for superinrv 
posing the generated sub image on a main im- 
age, comprising: 

input means for Inputting the sub image 
50 data packet Including: 

a sub image data portion for storing sub 
image data for displaying the sub image infonfna- 
tion; 

a header portion for storing sub image 
55 header data for assigning at least a display posi- 
tion, a display size, and a display color of the sub 
image data; and 

a control data portion for storing control 
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data for changing the display position, the display 
size, and the display color of the sub iniage data 
assigned by the sub image header data for each 
display unit of the main image; and 

means for controlling the sub image data 
stored in the sub image data portion correspond- 
ing to the sub image header data stored in said 
header portion and/or the control data stored in 
said control data portion so as to generate the 
sub image data. 
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FIG. 20 
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FIG. 25 
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